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In  order  to  improve  operator  performance  and  understanding  within  remote  environment, 
a  vision-based  virtual  forced  guidance  control  methodology  for  tele-robotic  system  is  pre¬ 
sented.  The  remote  operation  of  the  construction  robot  is  achieved  by  manipulating  the 
graphic  robot  in  a  virtual  environment.  Based  on  binocular  vision,  the  ground  surface  is 
modeled  as  an  elevation  map,  and  the  task  objects  are  recognized  from  video  images 
and  reconstructed  using  the  Power  Crust  algorithm.  The  virtual  guidance  forces  consisting 
of  a  pair  of  attractive  force  and  repulsive  force  from  the  objects  and  obstacles  are  used  to 
enhance  the  multi-task  manipulation  of  the  tele-robotic  system. 

©  2013  Elsevier  Ltd.  All  rights  reserved. 


1.  Introduction 

Tele-operation  technologies  have  been  widely  used  in  tele-robotic  systems  in  hazardous  environments,  such  as  outer 
space,  deep  oceans,  and  nuclear  plants.  In  conventional  remote  operation  systems,  there  are  limitations  on  the  number  of 
installed  sensors  and  the  volume  of  transmitted  data.  As  a  result,  limited  information  is  available  to  the  operator  on  site 
and  the  task  efficiency  is  significantly  influenced  [1,2]. 

Virtual  reality  technology  provides  operator  with  the  intuitive  cue  for  the  perception  of  the  working  fields  at  the  remote 
site,  which  could  potentially  enhance  the  system  with  greater  efficiency  and  facility  [2  .  Moreover,  it  provides  the  operator 
with  a  “live”  virtual  representation  of  the  scene  instead  of  the  delayed  video  images  [3].  For  example,  Qingping  Lin  intro¬ 
duced  a  virtual  tele-presence  interface  operation  approach,  which  greatly  improved  the  efficiency  of  tele-presence  operation 
of  underwater  robots  by  providing  the  operation  with  a  3-D  virtual  representation  of  the  underwater  environment  [4].  How¬ 
ever,  the  virtual  environment  is  commonly  built  upon  a  “beforehand”  modeling  method  with  hypotheses  that  the  position, 
size  and  contour  of  task  objects  have  been  known  in  advance.  Besides,  the  operator  has  to  judge  the  relative  posture  between 
the  end-effector  of  robot  and  the  task  object  with  vision-only  information.  It  is  not  applicative  for  unstructured  environment 
and  increases  the  operator’s  workload  and  stress. 

Force  or  haptic  feedback  is  proved  to  be  a  valid  method  for  improving  the  safety  and  efficiency  of  tele-robotic  system  [5]. 
As  psychophysiology  reveals  that  tactile  stimulus  incorporated  with  vision  data  brings  about  faster  response  time  compared 
to  vision-only  stimulus  [6,7],  many  more  studies  on  vision-force  hybrid  servoing  and  impedance  control  strategy  show  the 
trend  of  incorporating  haptics  and  vision  with  manipulation.  However,  in  these  cases,  the  high  cost  of  force  sensors  plays 
critical  roles,  and  the  direct  relationship  between  visual  data  and  haptic  force  is  less  studied  [6].  Moreover,  the  direct  force 
signals  which  have  time  delay  and  noise  in  transmission  are  only  generated  when  the  robot  interacts  with  the  environment 
[8-10].  According  to  the  feedbacks  of  tele-operation,  operators  are  more  concerned  about  the  relative  posture  between  robot 
and  environment.  The  force  sensors  could  not  provide  valid  information  under  the  free-moving  phase  of  robot.  A  virtual 
collision-preventing  force  was  generated  according  to  the  relative  position  between  robot  gripper  and  the  environment 
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obtained  with  a  trinocular  camera  [1 1,12].  Similarly,  haptic  feedback  was  computed  from  the  range  information  obtained  by 
a  sonar  array  attached  to  the  robot,  and  delivered  to  an  operator’s  hand  via  a  haptic  probe  [13,14].  However,  in  those  men¬ 
tioned  methods  the  haptic  feedback  was  mainly  used  for  collision  prevention,  instead  of  improving  operational  efficiency  of 
tele-robotic  system. 

To  improve  the  operational  safety  and  efficiency  of  tele-robotic  system,  a  force  guidance  control  methodology  is  pre¬ 
sented  in  this  paper.  The  visual  data  captured  by  a  binocular  vision  system  is  utilized  not  only  for  generating  the  virtual  envi¬ 
ronment  of  working  fields  in  real  time,  but  also  for  providing  the  operator  with  force  guidance  data  which  includes  the 
attractive  force  from  the  task  object  and  the  repulsive  force  from  the  obstacles.  The  human  decision-making,  the  safety 
and  the  manipulation  performance  of  tele-robotic  system  are  improved  by  combining  visual  data  with  the  virtual  force 
feedback. 

The  rest  of  the  paper  is  structured  as  follows:  in  Section  2,  we  describe  structure  of  tele-robotic  system  with  force  guid¬ 
ance,  involving  a  bilateral  section-master  and  slave.  Then  we  describe  the  3-D  reconstruction  of  task  object  and  ground  sur¬ 
face  by  the  stereo  vision  camera  in  Section  3.  Based  on  the  artificial  potential  field  theory,  Section  4  shows  the  Generation  of 
guiding  forces,  including  the  attractive  force  from  task  object  and  repulsive  force  from  obstacles.  In  Section  5,  comparison 
experiments  are  shown  under  the  circumstances  of  force  guidance  and  other  situations.  Finally,  Section  6  concludes  the 
paper. 


2.  Architecture  of  the  tele-robotic  system  with  force  guidance 

Fig.  1  shows  the  block  diagram  of  the  tele-robotic  system  with  vision-based  force  guidance.  The  system  involves  a  bilat¬ 
eral  master  and  slave.  The  master  is  controlled  by  an  operator  and  mainly  consists  of  two  force  feedback  joysticks  and  a 
screen  with  computer  graphics  (CG).  For  force  feedback  joysticks,  Microsoft  SiderWinder2  force  feedback  joysticks  are 
adopted,  which  are  capable  of  delivering  around  100  different  forces  and  16  programmable  buttons  (8  action  buttons  plus 
8-direction  hat).  The  slave  is  composed  of  a  construction  robot  and  a  stereo  vision  camera  called  “Bumblebee”.  The  construc¬ 
tion  robot  has  four  hydraulic  actuators  controlled  by  four  servo  vales  through  a  control  computer  (PCI),  and  the  “Bumble¬ 
bee”,  which  is  a  calibrated  camera  system  as  a  product  of  Point  Grey  Research  Inc.,  is  mounted  on  top  of  the  robot  and  its 
optical  axis  is  perpendicular  to  the  floor.  This  camera  provides  real  time  stereo  image  capture  for  applications  such  as  track¬ 
ing,  building  virtual  reality  model,  human  machine  interface  and  mobile  robotics.  It  can  accurately  measure  the  distance  of 
the  robot  to  the  object  in  its  field  of  view  at  a  speed  of  up  to  20  frames  per  second. 

By  means  of  the  stereo  camera  “Bumblebee”,  the  task  objects  in  images  are  recognized  and  their  contour  information  is 
extracted  with  a  feature  point  based  on  stereo  matching  algorithm.  Moreover,  the  terrain’s  height  information  of  working 
field  is  also  calculated  according  to  the  dense  stereo  matching  results  of  binocular  vision.  Therefore,  the  task  objects  and 
the  terrain  are  able  to  be  rendered  in  the  3-D  virtual  environment  on  a  graphic  computer  (PC2)  with  this  information. 

The  operator  performs  remote  operation  of  the  construction  robot  by  the  joysticks  with  the  assistance  of  the  3-D  virtual 
scene  in  front.  Operational  signals  for  the  servo  valves  of  the  robot  are  then  generated  by  PCI  which  processes  the  opera¬ 
tional  information  from  the  joysticks.  The  displacements  of  hydraulic  cylinders  of  the  robot  are  detected  by  on-site  displace¬ 
ment  sensors  mounted  to  hydraulic  cylinders  of  the  construction  robot  and  input  to  PC2,  inducing  the  corresponding 
movement  of  the  graphic  robot  in  the  virtual  world.  With  the  information  of  the  relative  posture  between  robot  and  envi¬ 
ronment,  the  virtual  forces,  including  the  attractive  force  from  the  task  object  and  the  repulsive  force  from  the  obstacles  or 
ground  surface,  are  calculated  in  PC2  and  fed  back  to  the  operator.  With  the  assistance  of  these  forces,  the  operator  is  able  to 
control  the  robot  approaching  target  quickly  and  avoiding  collision  with  obstacles  automatically. 


Fig.  1.  Construction  tele-robotic  system  with  force  guidance. 
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3.  Vision-based  3-D  reconstruction 

In  this  system,  the  stereo  camera  “Bumblebee”  has  a  built-in  vision  processing  unit  and  is  able  to  perform  fast  coordinate 
calculation  for  given  pixels  in  the  camera  image.  To  extract  3-D  positions  with  stereo  vision,  two  key  tasks  are  involved:  (1) 
Identify  the  corresponding  pixel  in  the  image  from  one  camera  for  a  given  pixel  in  the  other  camera.  (2)  Determine  a  ray  in 
the  3-D  space  associated  with  each  of  the  corresponding  pair  of  pixels.  In  this  paper,  the  camera  is  built  upon  a  pin-hole  im¬ 
age  model,  which  is  accurate  enough  for  distance  measurement.  In  order  to  improve  the  calibration  of  the  cameras,  a  feature 
in  the  image  captured  by  one  camera  is  required  to  lie  along  a  straight  line  in  the  other  image. 

The  virtual  environment  is  developed  with  the  open  source  OpenGL  library  in  Visual  C++.  It  aims  to  simulate  the  move¬ 
ment  of  robot  and  reconstruct  the  3-D  shape  for  task  objects  in  VR  world.  It  also  provides  the  operator  with  the  graphic  inter¬ 
face  for  the  accomplishment  of  tele-operation  task.  To  build  and  maintain  the  coordinate  relationship  between  each  moving 
part  of  the  graphic  robot,  a  scene-graph  representation  is  adopted,  in  which  each  joint  is  treated  as  a  graphic  entity  node  and 
attached  to  its  parent  node  in  a  tree  structure.  The  movement  of  parent  node  will  cause  the  corresponding  movement  of  all  of 
its  child  nodes.  During  the  simulation,  all  nodes  in  the  scene  are  traversed  and  updated  in  every  frame. 

3.1.  Reconstruction  of  task  object 

The  3-D  reconstruction  of  the  task  object  is  based  on  the  Power  Crust  algorithm  15],  which  guarantees  to  produce  geo¬ 
metrically  correct  approximations  of  the  objects  even  when  the  sample  points  fail  to  be  sufficiently  dense  [16].  The  idea  of 
the  Power  Crust  is  that  the  medial  axis  transform  (MAT)  of  the  object  is  first  approximated  with  a  set  of  samples.  The  MAT 
estimation  is  a  subset  of  the  Voronoi  vertices  called  poles.  The  object  approximation  is  then  obtained  with  a  set  of  finite  polar 
balls  centered  on  each  pole  of  the  estimated  MAT,  with  the  radius  corresponding  to  the  weight  of  the  pole.  The  object  surface 
could  be  produced  from  the  diagram  of  weighted  poles. 

To  concentrate  on  the  generation  of  guiding  force,  obvious  differences  in  the  color,  grayscale  and  texture  between  task 
object  and  ground  surface  are  supposed  in  the  system.  The  procedure  of  recognition  and  3-D  reconstruction  of  task  objects 
are  demonstrated  as  follows: 

(1 )  A  “Bumblebee”  camera  captures  a  video  image  (as  shown  in  Fig.  2(a))  in  its  field  of  view  and  measures  the  distances  to 
task  objects. 

(2)  Generate  the  binary  image  B0bj  for  the  task  objects  region  and  the  binary  image  Brob  for  the  robot  shadow  region  by 
applying  color,  texture,  and  height  threshold  to  the  input  images. 

(3)  Dilation  and  erosion  methods  are  applied  on  the  binary  image  so  as  to  eliminate  very  small  holes  caused  by  camera 
noise. 

(4)  Make  labels  i  (i=  1, 2,  3. . .)  for  the  isolated  regions  on  the  B0bj  and  remove  small  area  region  whose  size  is  little  than 
10  x  10  cm;  Also,  make  labels  j  (j  =  -1,  -2,  -3. . .)  for  the  Brob  and  only  keep  the  labeled  regions  with  maximum  area. 

(5)  Incorporate  the  B0bj  and  Brob  into  one  binary  image,  in  which  the  task  objects  region  could  be  distinguished  from  the 
robot  shadow  region  by  their  positive  or  negative  labeling  values. 

(6)  Extract  the  contours  C0bj  and  Crob  for  the  task  objects  region  and  the  robot  shadow  region  respectively  through  the 
point  splitting  and  merging  algorithm,  as  shown  in  Fig.  2(b). 

(7)  The  Crob  region  is  treated  as  polygon  and  used  to  detect  whether  the  task  objects  are  hidden  by  the  robot  or  not,  based 
on  the  collision  information  between  the  Crob  polygons  and  the  C0bj  polygons.  If  the  task  object  is  hidden,  then  the 
image  feature  of  C0bj  is  retrieved  from  previous  frame. 


(a)  Camera  image 


(b)  Contour  points 


Fig.  2.  Recognition  of  working  field. 
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Fig.  3.  3-D  shape  of  task  object. 


(8)  The  coordinates  of  all  the  points  within  CG bj  region  are  calculated,  and  their  3-D  point  clouds  are  generated.  With  the 
input  of  these  point  clouds,  the  surface  of  the  task  object  is  obtained  through  the  Power  Crust  algorithm,  as  shown  in 

Fig.  3. 

All  the  task  objects  in  the  virtual  world  are  assigned  with  certain  characteristics  of  mass,  density,  friction,  etc.  and  ex¬ 
pected  to  obey  the  Newton’s  physics  principle.  In  order  to  reduce  the  computational  complexity,  the  objects  in  current 
VR  application  are  treated  as  rigid  bodies  and  no  deformation  is  occurred  during  the  simulation.  The  dynamic  interaction 
between  virtual  objects  and  graphic  robot  is  realized  by  Nvidia  PhysX  SDK  2.8.3.  When  the  task  object  in  image  is  partially 
hidden  by  the  robot,  its  dynamic  behavior  in  VR  is  determined  by  the  physics  principle.  Otherwise,  the  position  of  task  object 
is  calculated  based  on  the  steps  above  and  updated  in  real-time. 

3.2.  Reconstruction  of  ground  surface 

The  “Bumblebee”  camera  provides  real-time  range  images  based  on  stereo  vision  technology,  which  allows  users  to  accu¬ 
rately  measure  the  distance  to  every  valid  pixel  in  an  image  and  to  generate  depth  map  fast  and  accurately. 

The  ground  surface  is  represented  by  an  elevation  map  coincident  with  the  x-y  plane  of  the  voxel  grid  [15].  The  height  of  a 
cell  in  the  elevation  map  is  set  to  the  average  height  of  the  points  found  in  the  bottom-most  occupied  voxel  in  the  voxel  array 
above  that  cell.  The  camera  image  and  the  calculated  elevation  map  are  shown  in  Fig.  4(a)  and  (b),  respectively. 

Due  to  the  potential  failure  of  the  pixel  matching  process,  some  height  map  cells  may  not  have  any  measurement  value. 
Therefore,  a  linear  interpolation  algorithm  is  applied  across  gaps  smaller  than  a  configurable  number  of  cells  in  X  and  Y  direc¬ 
tions  to  fill  small  holes  [17]. 

Like  other  approaches  based  on  stereo  matching,  the  “Bumblebee”  may  fail  to  produce  correct  3-D  points  along  the  dis¬ 
continuities  on  a  surface.  This  problem  becomes  more  noticeable  when  parts  of  working  field  or  task  object  were  hidden 
behind  the  robot.  A  5  x  5  Gaussian  filter  or  a  5  x  5  square  mask  median  filter  was  applied  for  the  low-pass  operation  to  pre¬ 
vent  sharp  transition  of  distance  in  the  depth  map.  Further,  in  order  to  enhance  the  realistic  appearance  of  the  ground  surface 
and  provide  operator  with  more  detailed  visual  cues,  a  texture  is  mapped  to  the  elevation  map  cells  with  the  video  imagery. 

4.  Generation  of  guiding  forces 

In  the  conventional  bilateral  servo  system,  the  tele-operative  manipulation  is  usually  performed  with  force  feedback, 
which  could  considerably  improve  the  user’s  ability  to  perform  complex  tasks.  However,  the  contact  force  detected  from  sen¬ 
sors  only  exists  when  the  robot  collides  with  environment.  Under  certain  conditions,  especially  in  a  high-speed  or  heavy¬ 
load  robot  system,  an  instant  contact  between  environment  and  robot  could  result  in  serious  consequences.  When  the  robot 
moves  freely  and  no  collision  occurs,  the  control  strategy  of  force  feedback  would  lose  its  role  since  the  force  sensors  could 
not  provide  valid  information  in  this  case. 

In  order  to  improve  the  safety  and  manipulation  performance  of  tele-robotic  systems,  a  vision-based  virtual  force  guid¬ 
ance  methodology  is  presented  in  this  paper.  The  virtual  forces,  which  include  the  attractive  force  from  task  object  and  repul¬ 
sive  force  from  obstacles,  are  built  upon  the  recognized  images  of  working  fields  using  binocular  vision  technique  rather  than 
generated  from  the  force  sensors.  Consequently,  the  kinesthesis  is  generated  from  the  virtual  forces  as  the  haptic  cue  and 
transmitted  to  the  operator  by  joysticks. 

According  to  the  artificial  potential  field  theory,  which  was  first  proposed  by  Khatib  [18]  and  originally  developed  for  an 
on-line  collision  avoidance  [19-21],  there  must  exist  one  path  that  can  lead  the  robot  move  towards  the  target  and  avoid 
obstacles  automatically  in  the  virtual  potential  field.  Therefore,  it  is  possible  to  develop  the  relationship  between  the  virtual 
force  and  the  potential  function,  which  is  the  key  part  of  the  new  force  guidance  strategy. 


T.  Ni  et  al.  /  Computers  and  Electrical  Engineering  39  (2013)  2135-2144 


2139 


(a)  Image  of  ground  surface 


o  o 

(b)  Calculated  height  map 


Fig.  4.  Reconstruction  of  ground  surface. 


The  robot  moves  under  the  influence  of  an  artificial  potential  field  V  defined  as  the  sum  of  an  attractive  potential  Vc  pull¬ 
ing  the  robot  towards  the  goal  configuration  y *  and  a  repulsive  potential  V0  pushing  the  robot  away  from  the  obstacles.  An 
artificial  force  F(y)  defined  as  F(y)  =  - VTyV  could  ensure  that  the  robot  reach  a  global  minimum  in  the  potential  field,  where  y 
is  the  vector  that  identifies  the  pose  of  the  end-effector  of  a  robot  in  the  extrinsic  workspace,  and  VJV  denotes  the  transpose 
of  the  gradient  vector  of  V  at  y. 

In  this  approach,  the  attractive  potential  field  VG  is  simply  defined  as: 

vc(y)  =  l(y-y,)2  (4.i) 

where  y  is  the  vector  that  identifies  the  pose  of  the  end-effector  of  a  robot  in  the  extrinsic  workspace,  and  y*  is  the  goal  con¬ 
figuration  of  robot. 

The  function  VG  is  positive  or  null  and  attains  its  minimum  at  y*.  Based  on  the  artificial  potential  field  theory,  the  attrac¬ 
tive  force  Fg  could  be  generated  as: 

Fc  =  -(y-y,)  (4.2) 


The  repulsive  potential  V0  is  defined  as: 


v0(y) 


0 


P^Po 
P>  Po 


(4.3) 


where  p  represents  the  distance  between  the  obstacle  and  the  end-effector  of  robot  along  its  velocity  direction,  p0  denotes 
the  influence  distance  of  this  potential  field.  r\  is  a  constant  which  determines  the  magnitude  of  repulsive  potential. 

The  function  V0  is  positive  or  null  and  reaches  to  a  maximum  at  the  boundary  of  the  obstacle,  but  it  will  reduce  to  zero 
when  the  robot  is  sufficiently  away  from  it.  The  repulsive  force  F0  could  be  generated  as: 

-1$  (4.4) 

0  P  >  Po 


where  v  is  the  normalized  velocity  vector  of  the  end-effector  of  robot. 
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In  this  system,  the  task  object  is  laid  on  the  ground  surface,  and  its  oriented  bounding  box  (OBB)  that  is  a  rectangular 
bounding  box  at  an  arbitrary  orientation  in  3-space  and  is  able  to  enclose  an  object  more  tightly,  as  is  oriented  to  the  lie 
of  the  object  22],  could  be  calculated  based  on  the  3-D  reconstructions  in  VR.  The  most  subvertical  axis  of  OBB  is  selected 
as  the  target  orientation  of  the  end-effector  of  the  robot.  Along  the  axis,  the  point  with  a  distance  of  (Z0/2  +  d)  to  the  center  of 
OBB  is  selected  as  the  target  position,  where  l0  is  the  length  of  OBB,  and  d  is  a  constant.  In  addition,  a  ray  is  sent  out  along  the 
velocity  direction  of  the  robot’s  end-effector  in  the  virtual  environment  and  its  distance  to  obstacles  or  the  intersection 
points  of  the  ground  surface  could  be  measured  by  the  function  interface  of  PhysX  SDK. 

The  guiding  force  feedback  to  the  joystick  is  given  by: 

Fm  —  ocFg  +  [>Fo  +  Fe  (4-5) 

where  the  scaling  factors  a  and  p  are  allowed  to  adjust  the  relative  influence  of  the  different  virtual  forces,  Fe  is  the  contact 
force  between  robot  and  environment  detected  from  force  sensors. 

Considering  the  importance  of  the  attractive  force  FG,  it  is  critical  to  achieve  an  acceptable  measurement  accuracy  of  the 
vision  system  about  the  position  of  task  object.  As  shown  in  previous  research  testimony,  in  the  1024  x  768  resolution,  the 
detection  error  of  the  vision  system  is  less  than  4  mm  when  the  object  is  2  m  far  from  Bumblebee  camera.  Furthermore,  this 
detection  error  approaches  the  index-diminishing  with  the  decrease  of  the  distance.  Therefore,  the  measurement  accuracy  of 
Bumblebee  could  assure  that  the  successful  information  is  transferred  from  the  workplace  to  the  virtual  force  in  the  process. 

During  the  free-moving  phase  of  manipulation,  the  effect  of  the  attractive  force  is  the  equivalent  of  adding  an  elastic  band 
between  the  target  and  the  end-effector  of  robot,  while  the  result  of  repulsive  force  is  just  like  putting  virtual  springs  along 
the  velocity  direction  of  the  robot.  With  the  assistance  of  the  virtual  forces,  the  operator  could  manipulate  the  robot 
approaching  the  target  quickly  and  avoiding  obstacles  automatically  simultaneously.  When  the  end-effector  reaches  its  tar¬ 
get  position  and  starts  to  grasp  the  object,  the  virtual  guiding  forces  will  become  zero  and  the  operator  will  feel  the  real 
grasping  force. 


5.  Experimental  result 

In  experiments,  operators  performed  specified  tasks  by  operating  the  construction  robot  by  the  joysticks.  The  construc¬ 
tion  robot  is  located  in  the  static  environment  which  is  unknown  beforehand.  By  the  approach  discussed  in  Chapter  3,  the  3- 
D  virtual  environment  was  generated.  The  operator  manipulated  the  on-site  robot  by  the  joysticks  with  the  assistance  of  the 
3-D  virtual  environment.  At  the  same  time,  the  operator  received  the  feedback  information  from  the  on-site  robot,  which 
caused  the  corresponding  movement  of  the  graphic  robot.  In  the  first  set  of  experiments,  as  shown  in  Fig.  5,  the  operator 
manipulated  the  robot  from  the  starting  point  in  which  the  robot  was  located  initially  to  target  object  in  area  C,  which  in¬ 
volves  some  simple  obstacles  in  the  working  fields,  and  the  following  manipulation  methods  were  executed. 

(1)  Method  one:  manipulation  in  the  virtual  environment  without  the  force  guidance. 

(2)  Method  two:  manipulation  in  the  virtual  environment  with  the  force  guidance. 

Fig.  6  shows  the  relative  distance  between  the  task  object  and  the  end-effector  of  the  robot  during  tele-operation.  The 
abscissa  shows  the  time  of  the  movement,  and  the  ordinate  shows  the  distance  in  the  corresponding  time.  Fig.  7  plots 
the  paths  that  the  robot  moves  from  the  start  point  to  the  final  destination.  The  red  dashed  line  represents  the  path  under 
the  manipulation  method  one,  while  the  black  solid  line  represents  the  path  under  the  manipulation  method  two. 


Fig.  5.  Task  area  for  evaluation  of  operation. 
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Fig.  7.  Motion  path  of  robot  toward  target. 


As  shown  in  Fig.  6,  force  guidance  control  methodology  can  significantly  shorten  the  operating  time  of  task  accomplish¬ 
ment  during  VR  assisted  tele-operation.  The  virtual  environment  could  provide  the  operator  with  distinct  recognition  of 
working  fields.  However,  parts  of  the  distance  information  may  be  lost  when  projected  onto  a  2-D  display.  Therefore,  the 
operator  still  needs  to  judge  the  relative  position  and  orientation  between  robot  and  task  object  in  the  2-D  image. 

In  addition,  considering  the  big  difference  between  the  master  manipulator  and  slave  robot  on  the  aspects  of  structure, 
size  and  manipulation  space,  it  is  necessary  to  regulate  many  times  the  control  commands  due  to  the  lack  of  experience  and 
feed-back  information  during  manipulation,  as  is  shown  in  Fig.  7.  As  a  result,  the  working  efficiency  of  tele-operation  is  re¬ 
duced,  and  the  psychological  stress  and  the  fatigue  of  decision-making  are  increased.  By  introducing  guiding  force  into  the 
VR  assisted  tele-operation,  the  relative  position  between  the  robot  and  the  task  object  is  fed  back  to  the  operator  in  terms  of 
force  and  haptic  hint,  which  is  significantly  beneficial  to  the  operator  when  manipulating  the  robot  approaching  the  target 
effectively,  rapidly,  smoothly  and  safely. 

Fig.  8  plots  forces  generated  on  joysticks.  The  abscissa  shows  the  time  of  the  movement,  and  the  ordinate  shows  the 
respective  force  on  joysticks  in  the  corresponding  time.  The  fix,  fly,  F2x  and  F2 y  represent  two-dimension  forces  on  joysticks 
respectively  (see  Fig.  8). 

While  the  end-effector  is  approaching  the  target,  the  guidance  forces,  which  include  the  attractive  and  repulsive  forces, 
are  generated  directly  based  on  visual  data  to  pull  or  push  the  operator  to  make  the  end-effector  approach  the  target  rapidly 
and  avoid  obstacles  automatically.  When  the  robot  is  located  in  the  initial  position  in  which  is  far  from  the  target,  the  virtual 
guidance  force  is  larger  and  is  capable  of  providing  the  operator  with  a  stronger  sense  of  touch.  As  the  robot  approaches  the 
target,  the  forces  tend  to  become  zero. 

In  the  second  set  of  experiments,  the  tele-operation  task  involved  the  transportation  of  objects  from  one  place  to  another, 
as  shown  in  Fig.  5. 
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Fig.  8.  Forces  generated  on  joysticks. 


(a)  Real  scene  before  grasping 


(b)  Real  scene  during  grasping 


(c)  VR  scene  before  grasping 


(d)  VR  scene  during  grasping 


Fig.  9.  Scene  of  experiments. 


Task: 

(1)  One  block  is  put  at  B  and  C  respectively. 

(2)  Block  at  C  is  transported  to  A. 

(3)  Block  at  B  is  stacked  on  top  of  the  block  at  A. 

Experiments  were  carried  out  under  three  various  conditions,  one  is  with  the  VR  plus  assisted  guiding  force,  one  is  with 
the  VR  only,  and  the  other  is  with  image  feedback  only.  The  real  and  virtual  scenes  for  manipulation  task  are  shown  in  Fig.  9. 
The  time  which  is  required  to  complete  the  task  under  each  condition,  the  time  and  the  contact  force  between  the  construc¬ 
tion  robot  and  the  ground  surface  are  measured. 

Fig.  10  shows  the  average  number  of  block  movements  that  the  tele-operation  tasks  completed  in  one  minute.  The  ab¬ 
scissa  shows  the  conditions  of  experiments,  and  the  ordinate  shows  task  efficiency  (objects/minute).  Larger  values  on  the 
ordinate  indicate  higher  task  efficiency.  With  the  assistance  of  guiding  force  and  VR  environment,  the  task  efficiency  in¬ 
creased  by  approximately  40%  in  comparison  with  the  VR-only  method,  and  almost  three  times  that  of  image-only  method. 
This  is  mainly  because  human  is  more  sensitive  to  force  or  haptic  hint  than  visual  hint.  With  the  virtual  force  guidance,  the 
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Fig.  10.  Experiment  of  efficiency. 


operator  does  not  need  adjust  the  end-effector  of  the  robot  redundantly  when  approaching  the  target  object.  In  the  mean¬ 
time,  the  operator  could  change  his  viewpoint  dynamically  to  achieve  the  best  view  position  and  to  avoid  being  hidden  by 
the  robot  arms  with  the  help  of  the  virtual  environment. 

Fig.  1 1  shows  the  average  time  and  contact  force  between  the  construction  robot  and  the  ground  surface.  The  abscissa 
shows  the  conditions  of  experiments,  and  the  left  ordinate  shows  the  time  of  contact  with  the  floor,  e.g.  tc  in  seconds,  while 
the  right  ordinate  shows  the  dimensionless  force  Fc  obtained  from  the  average  force  of  the  boom,  arm,  and  swing.  On  both 
the  right  and  left  axes,  larger  values  indicate  more  dangerous.  It  is  obvious  that  the  VR  plus  force  guidance  method  has  abso¬ 
lute  superiority  in  safety  among  all.  With  the  introduction  of  virtual  repulsive  force  from  obstacles,  the  robot  could  be 
pushed  away  before  collision  with  the  environment.  However,  with  2-D  image  feedback  only,  the  operator  feels  difficult 
to  judge  whether  the  robot  have  contacted  with  the  ground  or  not,  which  leads  to  a  higher  danger  coefficient  during  the 
operation. 


6.  Conclusion 

As  robots  play  more  diverse  and  important  roles  in  our  daily  lives  especially  in  hazardous  environment,  an  effective  inter¬ 
action  between  human  and  robots  as  well  as  appropriate  training  methods  concerning  the  safe  operation  of  the  robots  will 
be  required.  In  this  paper,  we  present  the  effective  way  of  guiding  human  operator  in  tele-manipulative  tasks  by  the  means 
of  virtual  force  guidance  and  virtual  reality  technology.  First  we  have  introduced  this  new  approach  with  respect  to  the  cor¬ 
relation  between  the  control  force  and  the  virtual  force  which  includes  a  pair  of  attraction  force  from  target  objects  and 
rejection  force  from  obstacles.  Then  a  virtual  reality  environment  is  established  to  couple  with  the  virtual  force  guidance 
to  enable  ease-of-use  for  human  operators  performing  common  manipulation  activities.  By  integrating  the  machine  intelli¬ 
gence  and  human  decision,  the  force  guidance  control  methodology  enhances  the  mixed-initiative  tele-operation  and  semi- 
autonomous  control  of  robotic  manipulators. 

Experimental  results  show  that  the  methodology  improves  the  tele-operation  performance  with  respect  to  efficiency, 
safety,  and  stress  reduction. 
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